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RELATED APPLICATION 

This application claims the benefit of U.S. Provisional Application No. 
60/41 1,561, filed on September 17, 2002. The entire teachings of the above application 
are incorporated herein by reference. 

BACKGROUND 

Multi hole off-axis apertures have been widely used in testing optical elements 
for nearly one hundred years, A mask with two off-axis apertures can also be used to 
correct or quantify defocusing and this technique, invented by the Jesuit astronomer 
Christoph Scheiner (1573-1650), has been used to focus telescopes for nearly 400 years. 
Others have used such a double aperture mask to create depth related image disparities 
in single exposed images in order to guide robots in three-dimensional object space. 
More recently, a three-aperture equivalent of the Hartmann mask has been applied to 
track particles in three-dimensions. These techniques derive quantitative depth 
information by sampling the optical wavefront according to the aperture positions. 
Since the mask with the off-axis apertures is stationary, such imaging systems are 
passive and their limitations are defined by the fixed baseline they apply. 
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SUMMARY 

The present invention is directed to active three-dimensional imaging systems 
adapted for two-dimensional optical apparatus. According to one embodiment, a three- 
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dimensional imaging system can include a relay lens subsystem with an off-axis 
rotating aperture placed at an exit pupil between an imaging or objective lens and an 
image recording device such as a CCD camera. According to an aspect, the relay lens 
subsystem includes a first lens group and a second lens group spaced apart with the 
5 aperture element disposed between the lens groups. The first lens group includes at 
least one field lens and the second lens group includes at least one focusing lens. 

According to another embodiment, the imaging system can include an off-axis 
rotating aperture located between field and aperture diaphragms of an illumination 
subsystem. In one embodiment, the rotating aperture is placed at the aperture 

10 diaphragm. According to an aspect, an optical apparatus is coupled to receive 

illumination fi^om the illumination subsystem along an illuminaUon path and an image 
recording device is coupled to the optical apparatus along the imaging path. In another 
aspect, the illumination subsystem fiirther includes a field diaphragm, an aperture 
diaphragm and a condenser spaced apart along the optical axis with the aperture 

15 element disposed between the field diaphragm and the aperture diaphragm. 

The rotating aperture allows adjustable non-equilateral spacing between images 
of out-of-focus object points to achieve higher spatial resolution, increased sensitivity 
and higher sub-pixel displacement accuracy than other systems. The three-dimensional 
imaging system may include rotation means for rotating the aperture element about the 

20 optical £ixis. 

The three-dimensional imaging system can be adapted with two-dimensional 
optical apparatus such as a microscope, telescope, endoscope or borescope. The 
imaging system can include an image processor for processing acquired images. 



25 BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, features and advantages of the invention will 
be apparent fi-om the following more particular description of preferred embodiments of 
the invention, as illustrated in the accompanying drawings in which like reference 
characters refer to the same parts throughout the different views. The drawings are not 
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necessarily to scale, emphasis instead being placed upon illustrating the principles of 
the invention. 

FIG. 1 is a schematic block diagram illustrating principles of a three- 
dimensional imaging system in accordance with one aspect of the present invention. 
5 FIG. 2A is a diagram showing details of a first embodiment of the system of 

FIG. 1. 

FIG. 2B is a diagram showing details of a second embodiment of the system of 

FIG. 1. 

FIG. 3 illustrates a rotating offset aperture for use in the system of FIG. 1 . 
10 FIG. 4 schematically illustrates the principle of the rotating offset aperture 

encoding depth information into circular image disparity. 

FIG. 5 is a schematic block diagram illustrating principles of a three- 
dimensional imaging system in accordance with another aspect of the present invention. 
FIG. 6A is a diagram showing an illumination system. 
15 FIG. 6B is a diagram showing an illumination subsystem with an off-axis 

aperture element in a first position of the system of FIG. 5. 

FIG. 6C is a diagram showing an illumination subsystem with an off-axis 
aperture element in a second position of the system of FIG. 5. 

FIG. 7A illustrates imaging of an in- focus object point at two positions of a 
20 rotating aperture element placed in the illumination path of a microscope. 

FIG. 7B illustrates imaging of an out-of-focus object point at two positions of a 
rotating off-axis aperture element placed in the illumination path of a microscope. 
FIG, 8 is a schematic cross-sectional diagram of a module. 
FIG. 9 is a double exposure image of a suspension of magnetic beads with a 
25 microscope having a rotating off-axis aperture element in the illumination path. 

DETAILED DESCRIPTION 

The present approach provides an active three-dimensional imaging system that 
includes an off-axis rotating aperture element placed either in the illumination path or in 
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the imaging path of an optical apparatus. The use of a rotating off-axis aperture element 
for three-dimensional imaging is disclosed in U.S. Application No. 09/616,606, filed 
July 14, 2000, and entitled "Method and Apparatus for High Resolution, Ultra Fast 3-D 
Imaging," the entire contents of which are incorporated herein by reference. 
5 FIG. 1 illustrates general principles of a three-dimensional imaging system that 

disposes an off-axis aperture in the imaging path. The system includes an optical 
apparatus 10, a relay subsystem 20 with an off-axis aperture element and a CCD camera 
30 disposed in optical alignment along an optical axis. The imaging system further 
includes a rotation mechanism 40 coupled to the relay subsystem 20 and an image 

10 processor 50 coupled to the CCD camera 30. 

The optical apparatus 10 can be a microscope, telescope, endoscope, borescope 
or other optical scope apparatus. The rotation mechanism 40 can be any type of 
electromechanical motor apparatus that can provide rotation of the off-axis aperture 
element. The image processor 50 can be implemented in a programmed general 

15 purpose computer. In other embodiments, the image processor 50 can be implemented 
in nonprogrammable hardware designed specifically to perform the image processing 
functions disclosed herein. The image processor can provide feedback to the rotation 
mechanism for control. 

FIG. 2 A illustrates details of a first embodiment of the optical apparatus 10 and 

20 relay subsystem 20 of FIG. 1. The optical apparatus can be, for example, a microscope 
or endoscope. The optical apparatus denoted 10b is shown as including a specimen 12 
and an objective lens group 14 that provides an image at image plane Ii . The relay 
subsystem 20b includes a field lens group 22 and a focusing or relay lens group 28 with 
an aperture element 24 disposed between the lens groups and aligned along the optical 

25 axis and placed at the exit pupil. The relay subsystem transfers the image at virtual 
image plane Ii to image plane I2 corresponding to a CCD element of CCD camera 30 
(FIG. 1). The magnification of the relay subsystem is on the order of M « 1 . The 
aperture element can also be referred to as a wavefront sampling aperture element. 
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In embodiments in which the optical apparatus includes an eyepiece as denoted 
for the optical apparatus 10a, the eyepiece serves the function of the field lens group 
and in that case the relay lens system may not include a field lens group as denoted 20a. 
As shown in FIG. 3, the aperture element 24 includes an off-axis opening 26. 
5 The off-axis opening 26 samples the blurred image of any out-of- focus point that results 
in a circular movement as the aperture rotates along a circle 24A around the optical axis 
through an angle denoted (p. The circular movement of the aperture encodes depth 
information into circular image disparity. The circle diameter represents the out-of- 
plane (z) and the center gives the in plane (x,y) coordinates of an object point relative to 

10 the reference (focal) plane. The phase difference between the movement of the aperture 
and an image spot indicates whether the point is behind or in front of the focal plane. 
The adjustable non-equilateral spacing between images permits higher spatial 
resolution, increased sensitivity and higher sub-pixel displacement accuracy in 
resolving object positions in the third (z) dimension. 

15 Referring again to FIG. 2 A, there are shown axial and chief ray bundles 13 and 

15, respectively. The axial ray bundle 13 includes ray traces 13a, 13b, 13c and the chief 
ray bundle 15 includes ray traces 15a, 15b, 15c. With the aperture element opening 26 
positioned at 90 degrees (FIG. 3), the ray traces 13b and 15b appear at the image plane 
I2. Likewise, with the aperture element opening 26 positioned at 270 degrees, the ray 

20 traces 13a and 15a appear at the image plane I2. 

FIG. 2B illustrates a second embodiment of the optical apparatus 10 and relay 
subsystem 20 of FIG. 1 . The optical apparatus can be, for example, a telescope. The 
optical apparatus denoted lOd is shown as including an objective lens group 54 that 
provides an image at image plane Ii . The relay subsystem 20d includes a field lens 

25 group 52 and a focusing or relay lens group 58 with an aperture element 55 (similar to 
aperture element 24 of FIGs. 2 A and 3) disposed between the lens groups and aligned 
along the optical axis. The relay subsystem transfers the image at virtual image plane Ii 
to image plane I2 corresponding to a CCD element of CCD camera 30 (FIG. 1). The 
magnification of the relay subsystem is on the order of M « 1 . 
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In embodiments in which the optical apparatus includes an eyepiece as denoted 
for the optical apparatus 10c, the eyepiece serves the function of the field lens group 
and in that case the relay lens system may not include a field lens group as denoted 20c, 
There are shown axial and chief ray bundles 17 and 19, respectively. The axial 
5 ray bundle 17 includes ray traces 17a, 17b, 17c and the chief ray bundle 19 includes ray 
traces 19a, 19b, 19c. With the aperture element opening positioned at 90 degrees (FIG. 
3), the ray traces 17b and 19b appear at the image plane I2. Likewise, with the aperture 
element opening positioned at 270 degrees, the ray traces 17a and 19a appear at the 
image plane I2. 

10 To understand the theory employed in the present imaging systems, FIG. 4 

illustrates the concept of measuring out-of-plane coordinates of object points by 
sampling the optical wavefi-ont, with an off-axis rotating aperture element, and 
measuring the defocus blur diameter. The system shown for illustrating the concept 
includes a lens 140, a rotating aperture element 160 and an image plane 18 A. The 

15 single aperture avoids overlapping of images from different object regions hence it 
increases spatial resolution. The rotating aperture allows taking images at several 
aperture positions and this can be interpreted as having several cameras with different 
viewpoints, which generally increases measurement sensitivity. The actively controlled 
aperture movement allows adjustable image disparity to increase accuracy of 

20 displacement detection. 

The aperture movement makes it possible to record on a CCD element a single 
exposed image at different aperture locations. In one approach to processing of the 
image, localized cross-correlation can be applied to reveal image disparity between 
image frames. There are several advantages to using cross-correlation rather than auto- 

25 correlation: reduced noise level, detection of zero displacement (in-focus points) and the 
absence of directional ambiguity, all of which are major problems of auto-correlation 
based processing. These advantages make higher signal-to-noise ratio, higher spatial 
and depth resolution, and lower uncertainty feasible. It should be understood that other 
processing approaches can be used that are not correlation based. 



As can be seen in FIG. 4, at least two image recordings on the image plane 18A 
at different angles of rotation of the aperture 160 are used to generate the measured 
displacement for target object 8 A. The separate images are captured successively as the 
aperture rotates to position #1 at time t and position #2 at time t+At. Note that At is 
generally negligible in relation to possible movement of the target object 8A. The 
recorded images can be processed using spatio-temporal processing. 

The rotation center of the image gives the proper in-plane object coordinates, 

Xo-^ (la) 



^ -yZo{L-f) 



where Xo,Yo are the in-plane object coordinates, f is the focal length of the lens 
objective, L is the depth of in-focus object points (focal plane), R is the radius of the 
circle along which the off-axis pupil is rotating, and d is the diameter of a circle along 
which the relevant out-of-focus point is moving on the image plane 1 8 A as the aperture 
is rotated. The magnitude of the pattern movement represents the depth information 
(Zo) measured from the lens plane. Zq can be evaluated from two Snell's lens laws for 
in-focus and out-of-focus object points and by using similar triangles at the image side, 

L IRfL ^ ^ 

The signal-to-noise ratio, relative error, and accuracy of detecting image pattern 
movement by image flow processing is influenced by the magnitude of the 
displacement vector being measured. For example, there is a trade-off between 
maximum detectable disparity and spatial resolution. This influence can be reduced and 
hence the dynamic range of displacement detection can be significantly improved by 
taking more than two images in non-equilateral aperture spacing. In order to remove 
possible ambiguity of image center detection, the rotation between the first and the third 



image is preferably 180 degrees while the intermediate recording can be adjusted 
according to the actual object depth. 

With the imaging system as described with respect to FIGs. 1, 2A and 2B, image 
correlation techniques can be used by the image processor 50 (FIG. 1) to provide ultra 
fast and super high resolution image processing as described in the above-referenced 
Application No. 09/616,606. For example, two image frames can be acquired. In 
particular, the aperture element is rotated such that the opening or off-axis aperture 26 
(FIG. 3) is at a first position offset from the optical axis. A first image frame is 
captured using the aperture at the first position. Likewise, a second image frame is 
captured with the aperture rotated through an angle such that the opening or exit pupil is 
at a second offset position. Once the frames are captured, the image processor 50 (FIG. 
1) can perform several techniques to correlate the information contained in the image 
frames. These techniques include sparse array cross-correlation, recursive correlation, 
correlation error correction and sub-pixel resolution processing. The sparse array image 
correlation approach is disclosed in U.S. Patent No. 5,850,485 issued December 15, 
1998, the entire contents of which are incorporated herein by reference. 

The advantage of the imaging system described above (FIGs. 1, 2 A and 2B) is 
its simplicity; however, sampling the defocus spot by an off-axis rotating aperture in the 
imaging path inherently has its own limitations. The most important is that it requires 
stepping down the aperture, which reduces optical resolution and requires stronger 
illumination. Lens aberration related systematic bias in the created image disparity also 
needs to be considered. 

These barriers can be overcome by placing the off-axis rotating aperture in the 
illumination path. FIG. 5 generally illustrates such an imaging system that includes an 
illumination subsystem 80 having an off-axis aperture, optical apparatus 100, and CCD 
camera 130. The imaging system also includes a rotation mechanism 140 and a image 
processor 150. 

FIG. 6A illustrates an illumination subsystem of a Kohler illumination 
configuration that includes field diaphragm 82, field lens group 83, aperture diaphragm 
86 and condenser 88. The right side illustrates the illumination light path and the left 
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side illustrates the image forming path. Illumination 96 ray traces include bundles 81, 
85 and 87 from an illumination source (not shown) that illuminate specimen 90. In the 
image forming path, ray bundles 91a, 91b; 93a, 93b; 95a, 95b correspond to respective 
points on the specimen 90. 
5 FIGs. 6B and 6C illustrate details of an embodiment of the illumination 

subsystem 80 that incorporates an off-axis rotating aperture element 84. In particular, 
the aperture element 84 is disposed between the field and the aperture diaphragms, here 
shown as being positioned in close proximity to the aperture diaphragm 86. The use of 
the off-axis aperture element creates oblique illumination. In FIG. 6B, with the aperture 

10 element in a first position (e.g., 90 degrees - FIG. 3), ray bundle 81 passes through the 
off-axis opening of the aperture element to the specimen 90 in the illumination light 
path. In the image forming path, ray traces 91a, 93a, 95a are the corresponding ray 
traces that are selected with this first position. Likewise, in FIG. 6C, with the aperture 
element in a second position (e.g., 270 degrees - FIG. 3) ray bundle 87 passes through 

1 5 the off-axis opening of the aperture element in the illumination path and ray traces 91b, 
93b, 95b are selected in the image forming path. 

The aperture element can be rotated by means of the rotation mechanism 140 
(FIG. 5) which may be of the general type described above for the rotation mechanism 
40 (FIG. 1). 

20 An in-focus object point at two positions of the rotating aperture placed in the 

illumination path is shown in FIG. 7A with respective oblique illumination light 96A, 
96B. Movement of the off-axis rotating aperture results in movement of all out-of- 
focus image points, as schematically illustrated in FIG. 7B. Although the applied off- 
axis rotating aperture gives weaker illumination intensity, and reduces the specimen size 

25 that can be imaged, the system is free of all resolution and aberration related limitations. 
This approach is also preferred if the objective has very long depth-of- focus, since this 
creates very small image disparities limiting the sensitivity of depth measurements. 

FIG. 8 is a cross-sectional diagram illustrating an embodiment of a module 200 
for coupling between an optical apparatus at end 224 and an image recording device 

30 such as a CCD camera at end 226. The module includes a focusing or relay lens 212 
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aligned with eyecup (microscope or endoscope) 210. In embodiments for use with 
optical apparatus that do not have an eyepiece, the eyecup can be replaced with a field 
lens. An off-axis rotating aperture element 208 is positioned between the relay lens 212 
and the eyecup 210 in aperture housing 222. The aperture housing 222 with bearing 
5 206 is coupled to a C-mount housing 214. A brushless DC-motor including rotor 
elements 216, 218 and stator 220 is operable to rotate the aperture element 208. 
Rotation control is achieved through means of encoder disk 204 and encoder pickup 
202. 

Numerous imaging systems currently being used for medical and research 

10 applications can be readily adapted with the three-dimensional apparatus of the present 
approach to provide three-dimensional quantitative measurement, hi particular, 
microscopes can be adapted into 3-D imaging devices using a module that encompasses 
either the relay subsystem or the illumination subsystem featuring off-axis aperture 
element described herein. Such microscopes adapted in this marmer can fill the gap 

15 between simple and lower accuracy methods such as stereo microscopy and more 
sophisticated systems such as confocal microscopy. 

In one particular example applying the present approach to magnetic bead 
microrheometry, a Hght microscope with an off-axis rotating aperture in the 
illumination path was used to image a suspension of magnetic beads, and to measure 

20 both the in-plane and out-of-plane position coordinates. Microscopic paramagnetic 
beads are attached to the cell membrane or embedded in the cytoskeleton and 
manipulated by magnetic tweezers. The bead displacement and cellular responses are 
monitored in three-dimensions to study mechanical properties and mechanotransduction 
events in single live cells. The sparsely distributed magnetic beads were imaged along 

25 the fiiU rotation of the off-axis aperture, and their movements were tracked through the 
image frames. 

FIG. 9 shows a preliminary result of this measurement. Two images fi-om the 
recorded sequence are shown overlapped, indicating the depth related image disparity of 
the beads. The image is a double exposure of the suspension of 4.5 |im magnetic beads 
30 imaged through a 63x objective. The off-axis aperture in the illumination path of the 
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microscope was rotated by 180 degrees between exposures. The circles on the image 
give an idea of the trajectories of the beads at full aperture rotation. They do not look 
closed because the focus of the microscope was slightly changed during image 
recording; this resulted in changing image disparities depending on the distance of the 
5 object points measured from the actual focal plane. 

The three-dimensional apparatus of the present approach can also be adapted for 
use with standard endoscopes to allow surgeons to not only see tissue structures in two 
dimensions, but also to quantify their size and exact three-dimensional shape with high 
spatial resolution reconstructed images in real time and without needing to constantly 
10 move the endoscope tip. Other adaptations may include borescopes and telescopes. 

While this invention has been particularly shown and described with references 
to preferred embodiments thereof, it will be understood by those skilled in the art that 
various changes in form and details may be made therein without departing from the 
scope of the invention encompassed by the appended claims. 



